Repository navigation
TEST-203 docs: add the Simulations onboarding API guide - #1344
Draft
scott-lowe-vapi wants to merge 2 commits into
Draft
scott-lowe-vapi wants to merge 2 commits into
scott-lowe-vapi wants to merge 2 commits into
Conversation
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1526d-5544-7429-b82e-095836bfb9eb.docs.buildwithfern.com |
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Oct 19, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scott-lowe-vapi
force-pushed
the
scottlowe/test-203-onboarding-api-guide
branch
from
October 19, 2026 04:46
ef56047 to
b6e5fd4
Compare
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1527c-90fc-733c-869a-5f48d4798f2d.docs.buildwithfern.com |
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1527c-d56f-7645-8012-59af756e398c.docs.buildwithfern.com |
| - **Cannot make business logic decisions**: You provide the domain knowledge; Composer handles the technical configuration. | ||
| - **Cannot test every behavior**: Suites Composer generates run in chat, so they can't check voice-only behavior such as pronunciation, and [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations) aren't tested. Make test calls yourself for these. | ||
| - **Cannot deploy to production automatically**: You control when and how changes go live. | ||
| - **Daily limits on test generation**: Requirement drafts, suite generations, and suite edits share a free daily allowance per organization with the [Simulations onboarding API](/observability/simulations-generate-api#limits). When your organization reaches a limit, Composer tells you and doesn't retry. The limits reset at 00:00 UTC. |
There was a problem hiding this comment.
The subject is three nouns, and "with the API" comes at the end.
Suggested change
| - **Daily limits on test generation**: Requirement drafts, suite generations, and suite edits share a free daily allowance per organization with the [Simulations onboarding API](/observability/simulations-generate-api#limits). When your organization reaches a limit, Composer tells you and doesn't retry. The limits reset at 00:00 UTC. | |
| - **Daily limits on test generation**: Each organization gets a free daily allowance of requirement drafts, suite generations, and suite edits. Composer and the [Simulations onboarding API](/observability/simulations-generate-api#limits) share it. When you reach a limit, Composer tells you and doesn't retry. Limits reset at 00:00 UTC. |
| slug: observability/simulations-generate-api | ||
| --- | ||
|
|
||
| The Simulations onboarding API builds a [**simulation suite**](/observability/simulations-overview) for an assistant from its prompt, tools, and handoffs. It's the same flow [**Composer**](/composer#generate-tests-for-an-assistant) runs when you ask it to generate tests, exposed as three endpoints you can call from a script, a CI job, or a coding agent: |
There was a problem hiding this comment.
Suggested change
| The Simulations onboarding API builds a [**simulation suite**](/observability/simulations-overview) for an assistant from its prompt, tools, and handoffs. It's the same flow [**Composer**](/composer#generate-tests-for-an-assistant) runs when you ask it to generate tests, exposed as three endpoints you can call from a script, a CI job, or a coding agent: | |
| The Simulations onboarding API builds a [**simulation suite**](/observability/simulations-overview) for an assistant from its prompt, tools, and handoffs. It runs the same flow as [**Composer**](/composer#generate-tests-for-an-assistant). Call these three endpoints from a script, a CI job, or a coding agent: |
| You then run the suite with the existing [`POST /eval/simulation/run`](/observability/simulations-quickstart) endpoint. | ||
|
|
||
| <Note> | ||
| **Beta.** Private API keys can call these endpoints only for organizations enabled during the beta. For any other organization they return `404`. [Contact support](/support) to enable your organization. Composer and the Dashboard work without this. |
There was a problem hiding this comment.
Suggested change
| **Beta.** Private API keys can call these endpoints only for organizations enabled during the beta. For any other organization they return `404`. [Contact support](/support) to enable your organization. Composer and the Dashboard work without this. | |
| **Beta.** During the beta, these endpoints only work for organizations we've enabled. Other organizations get a `404`. [Contact support](/support) to enable yours. You don't need this to use Composer or the Dashboard. |
| - An HTTP client with a long timeout. Suite generation can take up to 3 minutes, so set a client timeout of at least 200 seconds. | ||
|
|
||
| <Warning> | ||
| Generated simulations mock every tool in the assistant's tool list that would reach an external system, and the response lists them in `mockedTools`. Tools that only an assistant hook calls aren't mocked, and the assistant's server messages, such as the end-of-call report, still go to its server URL. Tools you add to a simulation later, or mocks you turn off, run for real. See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). |
There was a problem hiding this comment.
Four ideas in two sentences (and it repeats Advanced). Bullets:
Suggested change
| Generated simulations mock every tool in the assistant's tool list that would reach an external system, and the response lists them in `mockedTools`. Tools that only an assistant hook calls aren't mocked, and the assistant's server messages, such as the end-of-call report, still go to its server URL. Tools you add to a simulation later, or mocks you turn off, run for real. See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). | |
| Generated simulations mock every tool in the assistant's tool list that would reach an external system. The response lists them in `mockedTools`. These still run for real: | |
| - Tools that only an assistant hook calls. | |
| - Server messages, such as the end-of-call report, which still go to the assistant's server URL. | |
| - Tools you add to a simulation later, or mocks you turn off. | |
| See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). |
| } | ||
| ``` | ||
|
|
||
| Review the requirements and answer the open questions. A requirement tagged `assumption` rests on a guess; one tagged `not-testable` describes something a chat simulation can't check, such as pronunciation. |
There was a problem hiding this comment.
Split at the semicolon:
Suggested change
| Review the requirements and answer the open questions. A requirement tagged `assumption` rests on a guess; one tagged `not-testable` describes something a chat simulation can't check, such as pronunciation. | |
| Review the requirements and answer the open questions. A requirement tagged `assumption` is based on a guess. A requirement tagged `not-testable` is something a chat simulation can't check, such as pronunciation. |
| | `destination-truncated` | The destination wasn't checked because the assistant has too many handoff destinations to check in time, or it didn't load in time. The handoff into it is mocked. | | ||
| | `transfer-external` | A `transferCall` can leave Vapi, to a phone number, a SIP address, or a destination your server chooses, so it is mocked. | | ||
| | `destination-name-unscoped` | A handoff names a destination assistant outside a squad, which can't be resolved at runtime, so that destination is never reached. The handoff isn't mocked. | | ||
| | `native-tool-shadowed` | A tool that normally runs as is (such as `endCall` or `transferCall`) shares its name with a mocked tool, so the mock answers it too. | |
There was a problem hiding this comment.
"A tool that normally runs as is" is awkward. Maybe: "A built-in tool, such as endCall or transferCall, has the same name as a mocked tool, so the mock answers it too."
|
|
||
| A retry that returns a saved suite sets `reused` to `true` and doesn't count toward your limits. To generate a new suite from a revised draft, use a new key. | ||
|
|
||
| **Edits.** Send a new `patchKey` for each edit. Replaying a key that was already applied changes nothing and returns the current suite, with `failedEditSimulationIds` listing what that edit couldn't rewrite. So you can retry a timeout or a `5xx` with the same key. A suite remembers its last 50 patch keys, so never reuse one. |
There was a problem hiding this comment.
This contradicts itself: it says to retry a timeout with the same key, then ends "so never reuse one".
Suggested change
| **Edits.** Send a new `patchKey` for each edit. Replaying a key that was already applied changes nothing and returns the current suite, with `failedEditSimulationIds` listing what that edit couldn't rewrite. So you can retry a timeout or a `5xx` with the same key. A suite remembers its last 50 patch keys, so never reuse one. | |
| **Edits.** Send a new `patchKey` for each edit, and the same key to retry it. Replaying a key that was already applied changes nothing. It returns the current suite, with `failedEditSimulationIds` listing what that edit couldn't rewrite. A suite remembers only its last 50 patch keys, so never reuse a key for a different edit. |
|
|
||
| ## Limits | ||
|
|
||
| Each organization has a free daily allowance per UTC day, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC. |
There was a problem hiding this comment.
"a free daily allowance per UTC day" says "daily" twice. Maybe: "Each organization has a free daily allowance, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC."
|
|
||
| | Status | When | What to do | | ||
| | -- | -- | -- | | ||
| | `400` | The body is invalid; `previousDraft` and `feedback` were not sent together; or the generated or edited suite uses something your organization's data-region policy doesn't allow, so it isn't saved. | Fix the request. | |
There was a problem hiding this comment.
This cell (and the 422 row below) lists several causes separated by semicolons and parentheses. Consider one row per cause, or <br /> between causes.
| | --- | --- | | ||
| | Every iteration fails | Review the transcript, then verify the evaluation's description, data type, comparator, and expected value. | | ||
| | A run stays queued or running | Check the assistant or squad configuration and its provider credentials. Fetch the run and its items to find a failure reason. | | ||
| | Creating a voice run returns `400 Target assistant would fail to start a call: <message>` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. Vapi checks this only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. | |
There was a problem hiding this comment.
Optional: "Vapi checks this only for voice runs…" → "Vapi runs this check only for voice runs with an inline assistant target. It doesn't check a saved assistantId, a squad, or a chat run."
stephenvapiai
left a comment
stephenvapiai
left a comment
Contributor
There was a problem hiding this comment.
A few edit-pass notes inline: a reminder about the TODO comment, some terminology and wording fixes, and one question about patch keys. Most are suggestion blocks you can apply directly.
|
|
||
| A request counts once generation starts, even if it later fails with a `502` or times out. | ||
|
|
||
| {/* TODO verify: whether these limits apply to Enterprise orgs or can be raised on request. The source only says "free per-org quota". */} |
Contributor
There was a problem hiding this comment.
Reminder to remove this TODO comment before the page is published.
| @@ -0,0 +1,371 @@ | |||
| --- | |||
| title: Generate a test suite with the API | |||
Contributor
There was a problem hiding this comment.
The subtitle and body call this a simulation suite, so the title should match.
Suggested change
| title: Generate a test suite with the API | |
| title: Generate a simulation suite with the API |
|
|
||
| <Steps> | ||
| <Step title="Draft requirements"> | ||
| Send the assistant ID. Vapi reads the assistant's prompt, tools, and handoffs, and drafts the behaviors it must get right as numbered requirements, plus a few open questions. |
Contributor
There was a problem hiding this comment.
Our docs avoid naming Vapi as the actor when the platform is doing the work.
Suggested change
| Send the assistant ID. Vapi reads the assistant's prompt, tools, and handoffs, and drafts the behaviors it must get right as numbered requirements, plus a few open questions. | |
| Send the assistant ID. The endpoint reads the assistant's prompt, tools, and handoffs, and drafts the behaviors it must get right as numbered requirements, plus a few open questions. |
| </Step> | ||
|
|
||
| <Step title="Generate the suite"> | ||
| Send the agreed requirements and an `idempotencyKey`. Use one key per agreed draft, and reuse it for every retry of that draft. The key makes retries safe: a retry with the same key never generates a second suite. See [**Idempotency and retries**](#idempotency-and-retries). |
Contributor
There was a problem hiding this comment.
Capitalize after a colon when a full sentence follows.
Suggested change
| Send the agreed requirements and an `idempotencyKey`. Use one key per agreed draft, and reuse it for every retry of that draft. The key makes retries safe: a retry with the same key never generates a second suite. See [**Idempotency and retries**](#idempotency-and-retries). | |
| Send the agreed requirements and an `idempotencyKey`. Use one key per agreed draft, and reuse it for every retry of that draft. The key makes retries safe: A retry with the same key never generates a second suite. See [**Idempotency and retries**](#idempotency-and-retries). |
| -d @suite.json | ||
| ``` | ||
|
|
||
| Vapi plans scenarios that cover the requirements, writes each scenario and its success criteria, mocks the tools, then saves the suite and assigns it to the assistant. The response describes the saved suite: |
Contributor
There was a problem hiding this comment.
Same as above: the endpoint is the actor here.
Suggested change
| Vapi plans scenarios that cover the requirements, writes each scenario and its success criteria, mocks the tools, then saves the suite and assigns it to the assistant. The response describes the saved suite: | |
| The endpoint plans scenarios that cover the requirements, writes each scenario and its success criteria, mocks the tools, then saves the suite and assigns it to the assistant. The response describes the saved suite: |
|
|
||
| A retry that returns a saved suite sets `reused` to `true` and doesn't count toward your limits. To generate a new suite from a revised draft, use a new key. | ||
|
|
||
| **Edits.** Send a new `patchKey` for each edit. Replaying a key that was already applied changes nothing and returns the current suite, with `failedEditSimulationIds` listing what that edit couldn't rewrite. So you can retry a timeout or a `5xx` with the same key. A suite remembers its last 50 patch keys, so never reuse one. |
Contributor
There was a problem hiding this comment.
"Remembers its last 50 patch keys, so never reuse one" reads as a contradiction right after saying a retry with the same key is safe. What happens if a key is reused for a different edit? Saying that here would tell readers why to avoid it.
|
|
||
| ## Limits | ||
|
|
||
| Each organization has a free daily allowance per UTC day, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC. |
Contributor
There was a problem hiding this comment.
"Daily" and "per UTC day" say the same thing, and the next sentence gives the UTC reset.
Suggested change
| Each organization has a free daily allowance per UTC day, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC. | |
| Each organization has a free daily allowance, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC. |
| | `429` | A daily limit or the concurrent-generation limit was reached. | See [**Limits**](#limits). | | ||
| | `500` | Vapi couldn't read the assistant, its tools, or the suite. | Retry. For generation and edits, retry with the same key. | | ||
| | `502` | The model failed to generate a usable result. | Retry. For generation and edits, retry with the same key. | | ||
| | `503` | Simulation generation is paused, or no caller personalities are configured. | Retry later. | |
Contributor
There was a problem hiding this comment.
The other Simulations pages call these AI tester personalities.
Suggested change
| | `503` | Simulation generation is paused, or no caller personalities are configured. | Retry later. | | |
| | `503` | Simulation generation is paused, or no AI tester personalities are configured. | Retry later. | |
| | `500` | Vapi couldn't read the assistant, its tools, or the suite. | Retry. For generation and edits, retry with the same key. | | ||
| | `502` | The model failed to generate a usable result. | Retry. For generation and edits, retry with the same key. | | ||
| | `503` | Simulation generation is paused, or no caller personalities are configured. | Retry later. | | ||
| | `504` | The request took too long. | Retry. For generation and edits, retry with the same key: a generation may still save. | |
Contributor
There was a problem hiding this comment.
Capitalize after a colon when a full sentence follows.
Suggested change
| | `504` | The request took too long. | Retry. For generation and edits, retry with the same key: a generation may still save. | | |
| | `504` | The request took too long. | Retry. For generation and edits, retry with the same key: A generation may still save. | |
| | --- | --- | | ||
| | Every iteration fails | Review the transcript, then verify the evaluation's description, data type, comparator, and expected value. | | ||
| | A run stays queued or running | Check the assistant or squad configuration and its provider credentials. Fetch the run and its items to find a failure reason. | | ||
| | Creating a voice run returns `400 Target assistant would fail to start a call: <message>` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. Vapi checks this only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. | |
Contributor
There was a problem hiding this comment.
Same actor point as on the API page.
Suggested change
| | Creating a voice run returns `400 Target assistant would fail to start a call: <message>` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. Vapi checks this only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. | | |
| | Creating a voice run returns `400 Target assistant would fail to start a call: <message>` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. This check runs only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Draft: merge when VapiAI/vapi#23422 (TEST-200) deploys. That PR publishes the onboarding endpoints and lists this guide as a merge gate. Linear: TEST-203. Stacked on #1342.
/observability/simulations-generate-api), added to the nav after Advanced. It covers:idempotencyKey, edit withpatchKey, run withIdempotency-KeyFailed to create target callfailure reason (VapiAI/vapi#23400).Sources are the open stack's diffs (#23400, #23407, #23418–#23422). The 524 in the retry list comes from the endpoint's own operation description.
Before merge:
fern api updatepulls the routes in.What's New blurb:
Testing Steps
fern check(5.151.1) passes: 0 errors. The 14 warnings were already on main.